Imaging apparatus with phase difference detecting element

ABSTRACT

In an imaging element in which a plurality of phase difference detection pixels and a plurality of normal pixels for imaging are two-dimensionally arranged in a horizontal direction and a vertical direction, the phase difference detection pixel includes a first phase difference pixel ZA and a second phase difference pixel ZB including opening portions for pupil separation at different positions in the horizontal direction. The first phase difference pixel ZA and the second phase difference pixel ZB are adjacently arranged to have the opening portions facing each other. RGB color filters are arranged in the plurality of normal pixels in the Bayer arrangement. The imaging element includes a normal pixel row in which only the normal pixel is arranged in the horizontal direction and a phase difference pixel row in which the first phase difference pixel ZA, the second phase difference pixel ZB, and one normal pixel are periodically arranged in the horizontal direction.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation application of U.S. patentapplication Ser. No. 17/020,003 filed on Sep. 14, 2020 which is aContinuation application of U.S. patent application Ser. No. 16/509,826filed on Jul. 12, 2019 (now U.S. Pat. No. 10,819,900, issued Oct. 27,2020). U.S. patent application Ser. No. 16/509,826 is a Continuation ofPCT International Application No. PCT/JP2018/008354 filed on Mar. 5,2018 claiming priority under 35 U.S.C § 119(a) to Japanese PatentApplication No. 2017-051519 filed on Mar. 16, 2017. Each of the aboveapplications is hereby expressly incorporated by reference, in theirentirety, into the present application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an imaging element and an imagingapparatus and particularly, to an imaging element including a phasedifference detection pixel and an imaging apparatus.

2. Description of the Related Art

Recently, a technology for arranging a first phase difference pixel anda second phase difference pixel having opening portions for pupilseparation at different positions as a phase difference detection pixelin an imaging element and performing phase difference AF based on aphase difference between two image signals obtained from the first phasedifference pixel and the second phase difference pixel has been widelyused in order to increase the speed of autofocus (AF) (JP2013-013006A).

The imaging element disclosed in JP2013-013006A has color arrangement ofcolor filters of red (R), green (G), and blue (B) corresponding to theBayer arrangement. The first phase difference pixel and the second phasedifference pixel are arranged instead of G pixels and B pixels in aspecific GB row of the Bayer arrangement in which pixels (G pixels)including the G filter and pixels (B pixels) including the B filter arealternately arranged in the horizontal direction.

An imaging element disclosed in WO2012/169318 has color arrangement ofcolor filters corresponding to Bayer arrangement in the same manner asthe imaging element disclosed in JP2013-013006A. A phase differencedetection pixel pair including the first phase difference pixel and thesecond phase difference pixel is disposed at discrete and periodicpositions in a square grid. Of each pixel in the square grid, the firstphase difference pixel is disposed at 2n+1 (n=1, 2, . . . ) intervals inthe horizontal and vertical directions, and the second phase differencepixel is disposed as a pair with a pixel that is separated by two pixelsfrom the first phase difference pixel and includes the color filterhaving the same color as the first phase difference pixel.

In the imaging element disclosed in WO2012/169318, a phase differencedetection pixel pair of R pixels, a phase difference detection pixelpair of G pixels, and a phase difference detection pixel pair of Bpixels are disposed as the phase difference detection pixel pair atdiscrete and periodic positions in the square grid. Thus, a first imageincluding the first phase difference pixel and a second image includingthe second phase difference pixel are images having a disparitycorresponding to the distance of a subject. That is, the imaging elementdisclosed in WO2012/169318 can capture a normal planar image and a solidimage (two images including an image of only the first phase differencepixel and an image of only the second phase difference pixel).

SUMMARY OF THE INVENTION

In the imaging element disclosed in JP2013-013006A, the phase differencedetection pixel (first and second phase difference pixels) is arrangedat high density. Thus, AF performance is favorable, but a problem arisesin that correction accuracy for the phase difference detection pixel isdecreased.

The correction of the phase difference detection pixel is performed by“average value interpolation” in which interpolation is performed usingthe weighted average value of a plurality of normal pixels (pixels otherthan the phase difference detection pixel) present around the in-focusphase difference detection pixel. However, in the imaging elementdisclosed in JP2013-013006A, since the phase difference detection pixelis arranged at high density in the horizontal direction (lateraldirection) of the imaging element, it is difficult to perform theaverage value interpolation (correction using the average value ofnormal pixels close in the horizontal direction to the phase differencedetection pixel) in a case where the subject is a lateral line, therebyresulting in a problem of being unable to perform the average valueinterpolation with high accuracy.

Meanwhile, in the imaging element disclosed in WO2012/169318, in orderto be able to capture the solid image, it is necessary to dispose eachof the phase difference detection pixel pair of R pixels, the phasedifference detection pixel pair of G pixels, and the phase differencedetection pixel pair of B pixels at discrete and periodic positions inthe square grid. Thus, the phase difference detection pixels of the pairin which the color filters having the same color are arranged areseparated by two pixels from each other.

In the disclosure of WO2012/169318, the phase difference detectionpixels of the pair in which the color filters having the same color arearranged are not used in the phase difference AF. In addition, while thephase difference detection pixel pair of R pixels, the phase differencedetection pixel pair of G pixels, and the phase difference detectionpixel pair of B pixels have a disparity in the horizontal direction(left-right direction) depending on the distance to the subject, thephase difference detection pixels of the pair in which the color filtershaving the same color are arranged are separated from each other by twopixels in the vertical direction. Thus, phase difference AF cannot beperformed with high accuracy.

The present invention is conceived in view of such a matter. An objectof the present invention is to provide an imaging element and an imagingapparatus having high correction accuracy for a phase differencedetection pixel and favorable AF performance and furthermore, suitablefor low power and high speed processing.

In order to achieve the object, the invention according to one aspect isan imaging element in which a plurality of phase difference detectionpixels and a plurality of normal pixels for imaging aretwo-dimensionally arranged in a first direction and a second directionorthogonal to the first direction. The phase difference detection pixelincludes a first phase difference pixel and a second phase differencepixel including opening portions for pupil separation at differentpositions in the first direction, and the first phase difference pixeland the second phase difference pixel are adjacently arranged to havethe opening portions facing each other. In the plurality of normalpixels, a first filter corresponding to a first color most contributingto obtaining a brightness signal and a plurality of second filtersrespectively corresponding to two or more colors other than the firstcolor are arranged in a first periodic color arrangement. The imagingelement includes a normal pixel row in which only the normal pixel isarranged in the first direction and a phase difference pixel row inwhich the first phase difference pixel, the second phase differencepixel, and the normal pixel are periodically arranged in the firstdirection. Only the normal pixel row is arranged in at least two rowsadjacent to the phase difference pixel row. In the phase differencepixel row, in a case where only the normal pixel extending in the seconddirection from a position at which the normal pixel is arranged isextracted, a color filter is arranged in the first periodic colorarrangement in the extracted normal pixel.

According to one aspect of the present invention, the phase differencedetection pixel (the first phase difference pixel and the second phasedifference pixel) and the normal pixel are periodically arranged in thephase difference pixel row. Only the normal pixel is arranged in atleast two rows adjacent to the phase difference pixel row. Thus, in thecase of generating a pixel value at a pixel position of the phasedifference detection pixel by interpolating a pixel value of asurrounding pixel, pixel values of the normal pixels of at least tworows adjacent to the phase difference pixel row and the pixel value ofthe normal pixel in the phase difference pixel row can be used. Thephase difference detection pixel can be more accurately corrected(interpolated). In addition, in the phase difference pixel row, in acase where only the normal pixel extending in the second direction froma position at which the normal pixel is arranged is extracted, the colorfilter is arranged in the first periodic color arrangement in theextracted normal pixel. Thus, an image including the normal pixel havingthe first periodic color arrangement can be obtained, and low power andhigh speed processing are achieved. Furthermore, since a pair of thefirst phase difference pixel and the second phase difference pixel isadjacently arranged to have the opening portions facing each other, aninterval between the pair of the first phase difference pixel and thesecond phase difference pixel is the minimum. Accordingly, a spatialsampling frequency of a phase difference can be maximized, and phasedifference AF for a subject having a high spatial frequency can beperformed more favorably (with higher accuracy) than that in a casewhere the pair of the first phase difference pixel and the second phasedifference pixel is separately arranged with the normal pixel interposedtherebetween.

An imaging apparatus according to another aspect of the presentinvention comprises an imaging element in which a plurality of phasedifference detection pixels and a plurality of normal pixels for imagingare two-dimensionally arranged in a first direction and a seconddirection orthogonal to the first direction; the phase differencedetection pixel includes a first phase difference pixel and a secondphase difference pixel including opening portions for pupil separationat different positions in the first direction, and the first phasedifference pixel and the second phase difference pixel are adjacentlyarranged to have the opening portions facing each other; in theplurality of normal pixels, a first filter corresponding to a firstcolor most contributing to obtaining a brightness signal and a pluralityof second filters respectively corresponding to two or more colors otherthan the first color are arranged in a first periodic color arrangement;the imaging element includes a normal pixel row in which only the normalpixel is arranged in the first direction and a phase difference pixelrow in which the first phase difference pixel, the second phasedifference pixel, and the normal pixel are periodically arranged in thefirst direction, only the normal pixel row is arranged in at least tworows adjacent to the phase difference pixel row, and in the phasedifference pixel row; and in a case where only the normal pixelextending in the second direction from a position at which the normalpixel is arranged is extracted, a color filter is arranged in the firstperiodic color arrangement in the extracted normal pixel; an imagingoptical system forming a subject image on a light-receiving surface ofthe imaging element; a phase difference detection unit that detects aphase difference between a first pixel value obtained from the firstphase difference pixel and a second pixel value obtained from the secondphase difference pixel of the phase difference pixel row of the imagingelement; and an autofocus control unit that controls the imaging opticalsystem based on the phase difference detected by the phase differencedetection unit.

According to another aspect of the present invention, the subject imagecan be formed on the light-receiving surface of the imaging element(phase difference AF can be performed) by controlling the imagingoptical system based on the phase difference between the first pixelvalue obtained from the first phase difference pixel of the phasedifference pixel row of the imaging element and the second pixel valueobtained from the second phase difference pixel. Particularly, since thefirst phase difference pixel and the second phase difference pixel ofthe imaging element are adjacently arranged to have the opening portionsfacing each other, and the interval between the pair of the first phasedifference pixel and the second phase difference pixel is the minimum,the spatial sampling frequency of the phase difference can be maximized,and the phase difference AF for the subject having a high spatialfrequency can be performed more accurately than in a case where the pairof the first phase difference pixel and the second phase differencepixel is separately arranged with the normal pixel interposedtherebetween.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises a mode switching unitthat switches between a first mode for generating a first image and asecond mode for generating a second image, a first image generation unitthat generates pixel values at pixel positions of the first phasedifference pixel and the second phase difference pixel of the phasedifference pixel row based on at least a pixel value of the normal pixelof the phase difference pixel row and pixel values of the normal pixelsof at least two normal pixel rows adjacent to the phase difference pixelrow and generates the first image including the pixel values at thepixel positions of the first phase difference pixel and the second phasedifference pixel in a case where the mode switching unit switches to thefirst mode, and a second image generation unit that extracts only thenormal pixel extending in the second direction from a position at whichthe normal pixel is arranged in the phase difference pixel row, andgenerates the second image composed of pixel values of a plurality ofthe extracted normal pixels in a case where the mode switching unitswitches to the second mode.

According to still another aspect of the present invention, in the caseof switching to the first mode, the pixel values at the pixel positionsof the first phase difference pixel and the second phase differencepixel of the phase difference pixel row can be generated based on atleast the pixel value of the normal pixel of the phase difference pixelrow and the pixel values of the normal pixels of at least two normalpixel rows adjacent to the phase difference pixel row, and the firstimage (first image of high resolution) including the pixel values at thepixel positions of the first phase difference pixel and the second phasedifference pixel can be generated. In the case of switching to thesecond mode, the normal pixel extending in the second direction from theposition at which the normal pixel is arranged in the phase differencepixel row can be extracted, and the second image (second image of lowresolution) composed of the pixel values of the plurality of extractednormal pixels can be generated. The second image is composed of only thenormal pixel, and the color filters are arranged in the first periodiccolor arrangement in the second image. Thus. it is not necessary tocorrect the phase difference detection pixel, and image processing ofthe first image and the second image in a subsequent stage can beimplemented in common. Low power and high speed processing are achieved.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that in a case where the first phasedifference pixel and the second phase difference pixel are set as anin-focus pixel, the first image generation unit generates a pixel valueat a pixel position of the in-focus pixel using a pixel value of a pixelaround the pixel position of the in-focus pixel.

It is preferable that the imaging element according to still anotheraspect of the present invention further comprises a signal gradientcalculation unit that calculates a signal gradient direction in which asignal gradient of the pixel around the pixel position of the in-focuspixel is the minimum in a case of setting the first phase differencepixel and the second phase difference pixel as the in-focus pixel andgenerating the pixel value at the pixel position of the in-focus pixel,in which the first image generation unit detects a plurality of pixelspresent in the signal gradient direction calculated by the signalgradient calculation unit with the pixel position of the in-focus pixelas a reference and having the same color as a color at the pixelposition of the in-focus pixel and generates the pixel value at thepixel position of the in-focus pixel by interpolating pixel values ofthe plurality of detected pixels.

According to still another aspect of the present invention, a pixel usedfor interpolating the in-focus pixel in correspondence with the signalgradient direction of the surrounding pixel of the pixel position of thefirst phase difference pixel or the second phase difference pixel(in-focus pixel as a correction target) is selected. Thus, thecorrection accuracy of the phase difference pixel can be increased.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that the signal gradient calculationunit calculates one of four directions including the first direction,the second direction and a third direction and a fourth directionbetween the first direction and the second direction as the signalgradient direction.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that the signal gradient calculationunit calculates one of eight directions including four directions of thefirst direction, the second direction and a third direction and a fourthdirection between the first direction and the second direction and fourdirections midway among the four directions as the signal gradientdirection. By calculating the signal gradient direction using moredetailed directions than four directions, the correction accuracy of thephase difference pixel can be further increased.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that in a case where the signalgradient direction is not calculated by the signal gradient calculationunit, the first image generation unit generates the pixel value at thepixel position of the in-focus pixel based on a pixel value of a pixelclosest to the pixel position of the in-focus pixel and having the samecolor as a color at the pixel position of the in-focus pixel, or pixelvalues of a plurality of pixels having the same color in aninterpolation range with the pixel position of the in-focus pixel as areference.

In the imaging apparatus according to still another aspect of thepresent invention, it is preferable that in each of the first phasedifference pixel and the second phase difference pixel, the first filteris arranged, or light in a wavelength range wider than a transmissionwavelength range of the first filter is incident, the imaging apparatusincludes a pixel value addition unit that adds pixel values of the firstphase difference pixel and the second phase difference pixel adjacentlyarranged to have the opening portions facing each other and generates anaddition pixel value at a pixel position between the first phasedifference pixel and the second phase difference pixel, and the firstimage generation unit uses the addition pixel value added by the pixelvalue addition unit as the pixel value of one of the surrounding pixels.

In a case where the pixel values of the pair of the first phasedifference pixel and the second phase difference pixel adjacentlyarranged to have the opening portions facing each other are added, thepair of the first phase difference pixel and the second phase differencepixel behaves as if the normal pixel is present therebetween. The firstimage generation unit uses the addition pixel value as the pixel valueof one of the surrounding pixels in the case of correcting the phasedifference pixel. Thus, the correction accuracy of the phase differencedetection pixel can be improved.

It is preferable that the imaging apparatus according to still anotheraspect of the present invention further comprises a gain interpolationinformation obtaining unit that sets the first phase difference pixel orthe second phase difference pixel as the in-focus pixel and obtains gaininterpolation information set for the pixel position of the in-focuspixel, in which the first image generation unit generates the pixelvalue at the pixel position of the in-focus pixel by gain interpolationbased on a pixel value of the in-focus pixel and the gain interpolationinformation obtained by the gain interpolation information obtainingunit. Approximately half of the intensity of light incident on thesurrounding normal pixel is incident on the first phase difference pixeland the second phase difference pixel. Thus, sensitivity is decreasedbelow that of the normal pixel. The “gain interpolation” is aninterpolation method of adjusting a signal level to that of the normalpixel by multiplying the pixel value of the phase difference detectionpixel by the predetermined gain interpolation information in order tosupplement the decrease in sensitivity of the phase difference detectionpixel. In the case of correcting the phase difference detection pixel,depending on an imaging scene and the like, it may be more appropriateto perform the gain interpolation than average value interpolation thatuses the surrounding pixel of the in-focus pixel. In that case, the“gain interpolation” is performed.

In the imaging element according to still another aspect of the presentinvention, it is preferable that the first periodic color arrangement isthe Bayer arrangement, and the phase difference pixel row isperiodically arranged with three pixels including a pair of the firstphase difference pixel and the second phase difference pixel and onenormal pixel as one cycle. In a case where the first periodic colorarrangement is the Bayer arrangement, by reading the pixels of each rowof the imaging element at a thinning-out rate of 1/3, only the normalpixel can be obtained. In the case of a mode in which reading isexecuted in a thinned-out manner, the correction process for the phasedifference pixel is not necessary, and low power and high speedprocessing can be achieved.

In the imaging element according to still another aspect of the presentinvention, it is preferable that the first filter is a G filtertransmitting light in a wavelength range of green, and the plurality ofsecond filters include an R filter transmitting light in a wavelengthrange of red and a B filter transmitting light in a wavelength range ofblue, the first periodic color arrangement has a basic arrangementpattern of 4×4 pixels adjacently arranged in the first direction and thesecond direction, in the basic arrangement pattern, 2×2 pixels includingthe G filter and 2×2 pixels including the R filter are adjacentlyarranged in the first direction, and 2×2 pixels including the B filterand 2×2 pixels including the G filter are adjacently arranged in thefirst direction, and the phase difference pixel row is periodicallyarranged with three pixels including a pair of the first phasedifference pixel and the second phase difference pixel and one normalpixel as one cycle.

That is, the first periodic color arrangement is not limited to theBayer arrangement and may be the above color arrangement having thebasic arrangement pattern of 4×4 pixels. In this case, in the phasedifference pixel row, by periodically arranging three pixels includingthe pair of the first phase difference pixel and the second phasedifference pixel and one normal pixel as one cycle, only the normalpixel can be obtained by reading the pixels of each row of the imagingelement at a thinning-out rate of 1/3. In the case of a mode in whichreading is executed in a thinned-out manner, the correction process forthe phase difference detection pixel is not necessary, and low power andhigh speed processing can be achieved.

According to the present invention, in the phase difference pixel row inwhich the phase difference detection pixel (the first phase differencepixel and the second phase difference pixel) and the normal pixel areperiodically arranged in the first direction, the first phase differencepixel and the second phase difference pixel are adjacently arranged tohave the opening portions facing each other. Thus, in the case ofgenerating the pixel value at the pixel position of the phase differencedetection pixel by interpolating the pixel value of the surroundingpixel, the pixel values of the normal pixels of at least two normalpixel rows adjacent to the phase difference pixel row and the pixelvalue of the normal pixel of the phase difference pixel can be used. Thephase difference detection pixel can be accurately corrected. Inaddition, in the phase difference pixel row, in a case where only thenormal pixel extending in the second direction from the position atwhich the normal pixel is arranged is extracted, the color filter isarranged in the first periodic color arrangement in the extracted normalpixel. Thus, an image including only the normal pixel having the firstperiodic color arrangement can be obtained, and low power and high speedprocessing are achieved. Furthermore, since the first phase differencepixel and the second phase difference pixel are adjacently arranged tohave the opening portions facing each other, the phase difference AF canbe performed more favorably than that in a case where the pair of thefirst phase difference pixel and the second phase difference pixel isseparately arranged with the normal pixel interposed therebetween.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view illustrating one example of an imagingapparatus.

FIG. 2 is a rear view of the imaging apparatus illustrated in FIG. 1.

FIG. 3 is a block diagram illustrating one example of an internalconfiguration of the imaging apparatus illustrated in FIG. 1.

FIG. 4 is a diagram illustrating a first embodiment of color filterarrangement and arrangement of phase difference detection pixels in animaging element.

FIG. 5 is a plan view schematically illustrating a pair of a first phasedifference pixel ZA and a second phase difference pixel ZB.

FIG. 6 is an enlarged main view illustrating configurations of the firstphase difference pixel ZA and the second phase difference pixel ZB.

FIG. 7 is a block diagram illustrating an embodiment of an interpolationprocessing unit in an image processing unit 24 illustrated in FIG. 3.

FIG. 8 is a diagram for describing average value interpolation for thephase difference detection pixels in the imaging element of the firstembodiment.

FIG. 9 is another diagram for describing the average value interpolationfor the phase difference detection pixels in the imaging element of thefirst embodiment.

FIG. 10 is a diagram illustrating a second embodiment of color filterarrangement and arrangement of phase difference detection pixels in animaging element.

FIG. 11 is a diagram for describing the average value interpolation forthe phase difference detection pixels in the imaging element of thesecond embodiment.

FIG. 12 is another diagram for describing the average valueinterpolation for the phase difference detection pixels in the imagingelement of the second embodiment.

FIG. 13 is a diagram illustrating the exterior of a smartphone as oneembodiment of the imaging apparatus.

FIG. 14 is a block diagram illustrating an internal configuration of asmartphone 100 illustrated in FIG. 13.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of an imaging element and an imagingapparatus according to the present invention will be described inaccordance with the appended drawings.

Imaging Apparatus

FIG. 1 and FIG. 2 are respectively a perspective view and a rear viewillustrating one example (digital camera) of the imaging apparatus. Animaging apparatus 10 is a digital camera that receives light passingthrough a lens by an imaging element, converts the light into a digitalsignal, and records the digital signal in a recording medium as imagedata of a still picture or a motion picture.

As illustrated in FIG. 1, in the imaging apparatus 10, an imaging lens12, a strobe 1, and the like are arranged on a front surface, and ashutter button 2, a power supply/mode switch 3, a mode dial 4, and thelike are arranged on an upper surface. As illustrated in FIG. 2, aliquid crystal monitor (liquid crystal display (LCD)) 30, a zoom button5, a cross button 6, a MENU/OK button 7, a playback button 8, a BACKbutton 9, and the like are arranged on the rear surface of the camera.

The imaging lens 12 is composed of a retractable zoom lens and iswithdrawn from the main body of the camera by setting an operation modeof the camera to an imaging mode by the power supply/mode switch 3. Thestrobe 1 radiates strobe light to a main subject.

The shutter button 2 is composed of a so-called stroke type switch oftwo stages including “half push” and “full push” and functions as animaging preparation instruction unit and also functions as an imagerecording instruction unit.

In a case where a still picture imaging mode is selected as the imagingmode and the shutter button 2 is subjected to the “half push”, theimaging apparatus 10 performs an imaging preparation operation ofperforming autofocus (AF)/auto exposure (AE) control. In a case wherethe shutter button 2 is subjected to the “full push”, the imagingapparatus 10 images and records a still picture.

In addition, in a case where a motion picture imaging mode is selectedas the imaging mode and the shutter button 2 is subjected to the “fullpush”, the imaging apparatus 10 starts recording a motion picture. In acase where the shutter button 2 is subjected to the “full push” again,the imaging apparatus 10 stops recording and enters a standby state.

The power supply/mode switch 3 functions as a power supply switch forswitching a power supply of the imaging apparatus 10 ON/OFF and alsofunctions as a mode switch for setting the mode of the imaging apparatus10. The power supply/mode switch 3 is arranged to be slidable among an“OFF position”, a “playback position”, and an “imaging position”. Thepower supply of the imaging apparatus 10 is switched ON by sliding thepower supply/mode switch 3 to the “playback position” or the “imagingposition”. The power supply of the imaging apparatus 10 is switched OFFby sliding the power supply/mode switch 3 to the “OFF position”. Slidingthe power supply/mode switch 3 to the “playback position” sets a“playback mode”. Sliding the power supply/mode switch 3 to the “imagingposition” sets the “imaging mode”.

The mode dial 4 functions as a mode switching unit that sets the imagingmode of the imaging apparatus 10. The imaging mode of the imagingapparatus 10 is set to various modes depending on a setting position ofthe mode dial 4. For example, various modes include the “still pictureimaging mode” (first mode) for imaging a still picture and the “motionpicture imaging mode” (second mode) for imaging a motion picture.

The liquid crystal monitor 30 displays a live view image at the time ofthe imaging mode and displays the still picture or the motion picture atthe time of the playback mode. The liquid crystal monitor 30 alsofunctions as a part of a graphical user interface by, for example,displaying a menu screen.

The zoom button 5 functions as zoom instruction means for providing azoom instruction and includes a tele button 5T for providing a zoominstruction to a telephoto side and a wide button 5W for providing azoom instruction to a wide angle side. In the imaging apparatus 10,operating the tele button 5T and the wide button 5W at the time of theimaging mode changes the focal length of the imaging lens 12. Inaddition, operating the tele button 5T and the wide button 5W at thetime of the playback mode enlarges and shrinks the image in playback.

The cross button 6 is an operation unit that inputs instructions in fourdirections of upward, downward, leftward, and rightward directions andfunctions as a button (cursor movement operation means) for selecting anitem from the menu screen or providing an instruction to select varioussetting items from each menu. A left/right key functions as a button(forward direction/backward direction forwarding) for frame forwardingat the time of the playback mode.

The MENU/OK button 7 is an operation button functioning as a menu buttonfor providing an instruction to display the menu on a screen of theliquid crystal monitor 30 and also functioning as an OK button forproviding an instruction to, for example, confirm and execute thecontent of selection.

The playback button 8 is a button for switching to the playback mode inwhich the imaged and recorded still picture or motion picture isdisplayed on the liquid crystal monitor 30.

The BACK button 9 functions as a button for providing an instruction tocancel an input operation and return to the immediately previousoperation state.

In the imaging apparatus 10 according to the present embodiment, thefunctions of the buttons/switches may be implemented by disposing andoperating a touch panel without disposing members specific to thebuttons/switches.

Internal Configuration of Imaging Apparatus

FIG. 3 is a block diagram illustrating an embodiment of an internalconfiguration of the imaging apparatus 10. The imaging apparatus 10records the captured image in a memory card 54. The operation of thewhole apparatus is managed and controlled by a central processing unit(CPU) 40.

An operation unit 38 such as the shutter button 2, the power supply/modeswitch 3, the mode dial 4, the tele button 5T, the wide button 5W, thecross button 6, the MENU/OK button 7, the playback button 8, and theBACK button 9 is disposed in the imaging apparatus 10. A signal from theoperation unit 38 is input into the CPU 40. The CPU 40 controls eachcircuit of the imaging apparatus 10 based on the input signal. Forexample, the CPU 40 performs drive control of the imaging element, lensdrive control, stop drive control, imaging operation control, imageprocessing control, recording/playback control of the image data, anddisplay control of the liquid crystal monitor 30.

In a case where the power supply of the imaging apparatus 10 is switchedON by the power supply/mode switch 3, power is supplied to each blockfrom a power supply unit, not illustrated, and the imaging apparatus 10starts to be driven.

An image of a luminous flux passing through the imaging lens 12, a stop14, a mechanical shutter 15, and the like is formed in an imagingelement 16 that is a complementary metal-oxide semiconductor (CMOS) typecolor image sensor. The imaging element 16 is not limited to a CMOS typeand may be a color image sensor of an XY address type or a chargecoupled device (CCD) type.

In the imaging element 16, multiple light-receiving elements(photodiodes) are two-dimensionally arranged. A subject image formed ona light-receiving surface of each photodiode is converted into a signalvoltage (or charge) of an amount corresponding to the intensity of anincidence ray. The signal voltage is converted into a digital signalthrough an analog/digital (A/D) converter in the imaging element 16 andis output.

First Embodiment of Imaging Element

In the imaging element 16, color filters are arranged in a firstperiodic color arrangement, illustrated below, on a plurality of pixelscomposed of photoelectric conversion elements (photodiodes) that aretwo-dimensionally arranged in a first direction (horizontal direction)and a second direction (vertical direction) orthogonal to the firstdirection. In addition, in the imaging element 16, a plurality of phasedifference detection pixels and a plurality of normal pixels (pixelsother than the phase difference detection pixel) for imaging arearranged.

As illustrated in FIG. 5, the phase difference detection pixel includesan opening portion for pupil separation and is composed of a first phasedifference pixel ZA and a second phase difference pixel ZB havingopening portions at different positions in the horizontal direction. Apair of the first phase difference pixel ZA and the second phasedifference pixel ZB is adjacently arranged to have the opening portionsfacing each other. Details of the structures of the first phasedifference pixel ZA and the second phase difference pixel ZB will bedescribed below.

FIG. 4 is a diagram illustrating a first embodiment of color filterarrangement and arrangement of the phase difference detection pixels inthe imaging element 16.

As illustrated in FIG. 4, a first filter corresponding to a first color(green) and any of a plurality of second filters respectivelycorresponding to two or more colors (red and blue) other than green arearranged in a first periodic color arrangement in the plurality ofnormal pixels of the imaging element 16. The first periodic colorarrangement of the color filters of the imaging element 16 of the firstembodiment is the general Bayer arrangement. The first filter is a Gfilter transmitting light in a wavelength range of green. The pluralityof second filters include an R filter transmitting light in a wavelengthrange of red and a B filter transmitting light in a wavelength range ofblue.

In the imaging element 16 having the Bayer arrangement, normal pixelrows in which only the normal pixels are arranged in the horizontaldirection (row direction) include an RG row in which a pixel (R pixel)having the R filter and a pixel (G pixel) having the G filter arealternately arranged in the row direction, and a GB row in which the Gpixel and a pixel (B pixel) having the B filter are alternately arrangedin the row direction. In addition, the RG row and the GB row arealternately arranged in the vertical direction (column direction).

In addition, the G filter is arranged in each of the first phasedifference pixel ZA and the second phase difference pixel ZB of thepresent example. In the first phase difference pixel ZA and the secondphase difference pixel ZB, for example, light in a wavelength rangewider than the transmission wavelength range of the G filter may beincident without arranging the G filter.

In the imaging element 16, a phase difference pixel row in which thefirst phase difference pixel ZA, the second phase difference pixel ZB,and the normal pixel are periodically arranged in the row direction isdisposed in the GB row at an interval of a plurality of rows. Only thenormal pixels are arranged in at least two rows adjacent to the phasedifference pixel row.

In addition, in the phase difference pixel row of the present example,three pixels including the pair of the first phase difference pixel ZAand the second phase difference pixel ZB and one normal pixel areperiodically arranged with the three pixels as one cycle. Accordingly,in the phase difference pixel row, the G pixel and the B pixel arealternately arranged in the row direction at an interval of two pixels(the pair of the first phase difference pixel ZA and the second phasedifference pixel ZB).

While the phase difference pixel row of the present example is disposedin the GB row of the Bayer arrangement, the phase difference pixel rowis not for limitation purposes and may be disposed in the RG row.

In the imaging element 16 having the above configuration, as illustratedin FIG. 4, in a case where only the normal pixels extending in thesecond direction (column direction) from positions (positions indicatedby black star marks in FIG. 4) at which the normal pixels (G pixel or Bpixel) of the phase difference pixel row are arranged are extracted,color arrangement of the color filter of each pixel of the plurality ofextracted normal pixels forms the Bayer arrangement.

In addition, an image of one frame (frame image) constituting the motionpicture has a smaller image size than the still picture of full pixels,and the imaging element 16 is driven in a thinned-out manner in themotion picture imaging mode. Accordingly, low power and high speedprocessing can be achieved.

Even in a case where an image signal is read from the imaging element 16in a thinned-out manner at a certain row interval (rows indicated byblack circle marks in FIG. 4) as illustrated in FIG. 4, extracting onlythe normal pixels extending in the column direction from the positionsat which the normal pixels of the phase difference pixel row arearranged in the image corresponding to the image signal read in athinned-out manner results in the color arrangement of the color filterof each pixel of the plurality of extracted normal pixels forming theBayer arrangement.

That is, even in a case where the rows read from the imaging element 16in a thinned-out manner include the phase difference pixel row, it ispossible not to include the first phase difference pixel ZA and thesecond phase difference pixel ZB in the frame image constituting themotion picture. In addition, by reading the phase difference pixel row,phase difference AF based on the first phase difference pixel ZA and thesecond phase difference pixel ZB included in the phase difference pixelrow can be performed during imaging of the motion picture.

FIG. 5 is a plan view schematically illustrating the pair of the firstphase difference pixel ZA and the second phase difference pixel ZB.

As illustrated in FIG. 5, the first phase difference pixel ZA includesthe opening portion in the right half of the pixel, and the second phasedifference pixel ZB includes the opening portion in the left half of thepixel. That is, the opening portions of the pair of the first phasedifference pixel ZA and the second phase difference pixel ZB face eachother.

Accordingly, in a case where the image signals (pixel values) of thepair of the first phase difference pixel ZA and the second phasedifference pixel ZB are added, the added pixel value (addition pixelvalue) is almost equal to the pixel value of the normal pixel at thesame pixel position. In addition, the added pixel (addition pixel) canbe regarded as being midway between the pair of the first phasedifference pixel ZA and the second phase difference pixel ZB.

The normal pixel and the phase difference detection pixel have differentpixel characteristics. Thus, it is necessary to generate the stillpicture of full pixels after appropriately correcting the phasedifference detection pixel.

The correction of the phase difference pixel is performed by “averagevalue interpolation” in which interpolation is performed using theweighted average value of the pixel values of a plurality of normalpixels present around the in-focus phase difference detection pixel (thefirst phase difference pixel ZA or the second phase difference pixelZB).

The pixel value (in the present example, corresponds to the pixel valueof the G pixel of the normal pixel) of the addition pixel including thepair of the first phase difference pixel ZA and the second phasedifference pixel ZB can be used in a case where the pixel value of the Gpixel at the pixel position of the in-focus pixel of the first phasedifference pixel ZA or the second phase difference pixel ZB isinterpolated using the average value interpolation. Details of thecorrection of the phase difference value will be described below.

FIG. 6 is an enlarged main view illustrating configurations of the firstphase difference pixel ZA and the second phase difference pixel ZB.

As illustrated in FIG. 6, a light shielding member 16A is arranged onthe front surface side (microlens L side) of a photodiode PD of thefirst phase difference pixel ZA, and a light shielding member 16B isarranged on the front surface side of the photodiode PD of the secondphase difference pixel ZB. The microlens L and the light shieldingmembers 16A and 16B have a pupil separation function. In FIG. 6, thelight shielding member 16A shields the left half of the light-receivingsurface of the photodiode PD from light. Thus, the first phasedifference pixel ZA receives only a luminous flux passing on the leftside of an optical axis among luminous fluxes passing through an exitpupil of the imaging lens 12. In addition, in the present example, the Gfilter is arranged below the microlens L as a color filter CF.

The light shielding member 16B shields the right half of thelight-receiving surface of the photodiode PD of the second phasedifference pixel ZB from light. Thus, the second phase difference pixelZB receives only a luminous flux passing on the right side of theoptical axis among the luminous fluxes passing through the exit pupil ofthe imaging lens 12. By the microlens L and the light shielding members16A and 16B having the pupil separation function, the luminous fluxespassing through the exit pupil on the left and right sides are separatedand are respectively incident on the first phase difference pixel ZA andthe second phase difference pixel ZB.

Returning to FIG. 3, the image signal (image data) read from the imagingelement 16 at the time of imaging the motion picture or the stillpicture is temporarily stored in a memory (synchronous dynamic randomaccess memory (SDRAM)) 48 or is input into an AF processing unit 42, anAE detection unit 44, and the like through an image input controller 22.

The CPU 40 manages and controls each unit of the imaging apparatus 10based on the operation in the operation unit 38 and performs an AFoperation and an AE operation at all times during imaging (display) ofthe live view image and imaging (recording) of the motion picture.

The AF processing unit 42 functioning as a phase difference detectionunit is a part performing the phase difference AF process and detectsthe phase difference using the output signal of each of the first phasedifference pixel ZA and the second phase difference pixel ZB obtainedthrough the image input controller 22. Details of the detection of thephase difference by the AF processing unit 42 will be described below.

In a case where phase difference data indicating the phase difference isinput from the AF processing unit 42, the CPU 40 functions as a focalpoint adjusting unit that performs the phase difference AF based on thephase difference data. That is, the CPU 40 calculates a deviation amount(defocus amount) between a focus position of the imaging lens 12 and animage forming surface of the imaging element 16 based on the phasedifference data and moves a focus lens in the imaging lens 12 through alens drive unit 36 such that the calculated defocus amount becomes zero.The calculation of the defocus amount may be performed by the AFprocessing unit 42.

The AE detection unit 44 calculates the integrating accumulation of theimage data (for example, the pixel values of the G pixels of the wholescreen) obtained through the image input controller 22 or calculates theintegrating accumulation of the image data (pixel values of the Gpixels) differently weighted between a center portion and a peripheralportion of the screen and outputs the integrating accumulation value tothe CPU 40. The CPU 40 calculates the brightness (imaging exposure value(Ev value)) of the subject from the integrating accumulation value inputfrom the AE detection unit 44. In a case where the imaging mode is thestill picture imaging mode, the CPU 40 performs the above AF controlagain in a case where the shutter button 2 is subjected to a first stagepush (half push). In a case where the shutter button 2 is subjected tothe full push, the CPU 40 calculates the brightness (imaging Ev value)of the subject, decides the F-number of the stop 14 and a light exposuretime (shutter speed) of the mechanical shutter 15 based on thecalculated imaging Ev value, and images the still picture (exposurecontrol).

In a case where the imaging mode is the motion picture imaging mode, theCPU 40 starts imaging and recording (picture recording) the motionpicture in a case where the shutter button 2 is subjected to the fullpush. At the time of imaging the motion picture, the CPU 40 opens themechanical shutter 15, consecutively reads (for example, 30frames/second or 60 frames/second as a frame rate) the image data fromthe imaging element 16, consecutively performs the phase difference AF,calculates the brightness of the subject, and controls the shutter speed(a charge accumulation time by rolling shutter) by a shutter drive unit33 and/or the stop 14 by a stop drive unit 34.

The CPU 40 operates the zoom lens to advance and retract in the opticalaxis direction through the lens drive unit 36 in response to the zoominstruction from the zoom button 5 and changes the focal length.

In addition, ROM 47 is a read only memory (ROM) or an electricallyerasable programmable read-only memory (EEPROM) storing defectinformation related to the imaging element 16 and various parameters andtables used in image processing and the like. In the present example,the ROM 47 stores information related to the phase difference pixel row(including the pixel positions of the first phase difference pixel ZAand the second phase difference pixel ZB) and the normal pixel row ofthe imaging elements 16 and gain interpolation information, a leveladjusting coefficient, and the like described below.

The image processing unit 24 functioning as a first image generationunit and a second image generation unit reads non-processed image data(RAW data) that is temporarily stored in the memory 48 and is obtainedthrough the image input controller 22 at the time of imaging the motionpicture or the still picture. The image processing unit 24 performs anoffset process, a pixel interpolation process (interpolation process forthe phase difference detection pixel, a defective pixel, and the like),white balance correction, a gain control process including sensitivitycorrection, gamma-correction processing, demosaicing (referred to as a“demosaicing process”), a brightness and color difference signalgeneration process, a contour highlighting process, color correction,and the like on the read RAW data.

The image data processed as the live view image by the image processingunit 24 is input into a video random access memory (VRAM) 50.

The VRAM 50 includes an A region and a B region. In each of the A regionand the B region, image data representing an image of one frame isrecorded. In the VRAM 50, the image data representing the image of oneframe is alternately rewritten between the A region and the B region.The written image data is read from a region of the A region and the Bregion of the VRAM 50 other than a region in which the image data isrewritten.

The image data read from the VRAM 50 is encoded in a video encoder 28and is output to the liquid crystal monitor 30 disposed on the rearsurface of the camera. Accordingly, the live view image showing thesubject image is displayed on the liquid crystal monitor 30.

The image data (brightness data (Y) and color difference data (Cb) and(Cr)) processed as the still picture or the motion picture for recordingby the image processing unit 24 is stored in the memory 48 again.

A compression/expansion processing unit 26 performs a compressionprocess on the brightness data (Y) and the color difference data (Cb)and (Cr) processed by the image processing unit 24 and stored in thememory 48 at the time of recording the still picture or the motionpicture. In the case of the still picture, for example, the compressionis performed in the Joint Photographic Experts Group (JPEG) format. Inthe case of the motion picture, for example, the compression isperformed in the H.264 format. The compression image data compressed bythe compression/expansion processing unit 26 is recorded in the memorycard 54 through a media controller 52.

In addition, the compression/expansion processing unit 26 performs anexpansion process on the compression image data obtained from the memorycard 54 through the media controller 52 at the time of the playbackmode. For example, the media controller 52 records and reads thecompression image data in the memory card 54.

Phase Difference AF

In the case of performing the phase difference AF, the CPU 40functioning as an autofocus control unit outputs a read instruction forreading the image data of the phase difference pixel row in at least anAF region of the imaging element 16 to a sensor drive unit 32 and readsthe corresponding image data from the imaging element 16.

At the time of imaging and displaying the motion picture (including thelive view image), the CPU 40 obtains a thinning-out rate for reading theimage data from the imaging element 16 in a thinned-out manner. Thethinning-out rate may be a preset fixed value or may be able to beselected by a user from a plurality of thinning-out rates. For example,the optimal thinning-out rate can be set in connection with selection ofthe image size of the motion picture or selection of the frame rate. Therows read in a thinned-out manner include the phase difference pixelrow.

The CPU 40 outputs a read instruction indicating a thinning-out pattern(extraction pattern) corresponding to the thinning-out rate to thesensor drive unit 32 and reads the image data from the imaging element16 in a thinned-out manner.

The AF processing unit 42 extracts output data of the phase differencedetection pixel (the first phase difference pixel ZA and the secondphase difference pixel ZB) in the AF region from the read phasedifference pixel row and detects the phase difference between the outputdata (first pixel value) of the first phase difference pixel ZA and theoutput data (second pixel value) of the second phase difference pixelZB. For example, the phase difference is obtained from a shift amount inthe left-right direction between the first pixel value and the secondpixel value when the correlation between the first pixel value and thesecond pixel value of the pair of the first phase difference pixel ZAand the second phase difference pixel ZB is the maximum (when theintegrating accumulation value of the absolute value of the differencebetween the pixel values of the pair of the phase difference pixels isthe minimum).

A value obtained by correcting the obtained shift amount by a positionaldeviation in the horizontal direction between the pair of the firstphase difference pixel ZA and the second phase difference pixel ZB canbe calculated as the phase difference data. A method of calculating thephase difference is not limited to the above method, and various methodscan be applied.

Next, the CPU 40 calculates the deviation amount (defocus amount)between the focus position of the imaging lens 12 (imaging opticalsystem) and the image forming surface of the imaging element 16 based onthe phase difference data detected by the AF processing unit 42. Thecalculation of the defocus amount may be performed by the AF processingunit 42.

The CPU 40 performs the phase difference AF by moving the focus lens inthe imaging lens 12 through the lens drive unit 36 based on thecalculated defocus amount such that the defocus amount becomes zero.

In the imaging element 16, the pair of the first phase difference pixelZA and the second phase difference pixel ZB is adjacently arranged tohave the opening portions facing each other. Thus, the interval betweenthe pair of the first phase difference pixel ZA and the second phasedifference pixel ZB is the minimum. Accordingly, a spatial samplingfrequency of the phase difference can be maximized, and the phasedifference AF for the subject having a high spatial frequency can beperformed with higher accuracy than that in a case where the pair of thefirst phase difference pixel and the second phase difference pixel isseparately arranged with the normal pixel interposed therebetween.

The rows read from the imaging element 16 in a thinned-out manner at thetime of generating the motion picture can include the phase differencepixel row including the phase difference detection pixel (the firstphase difference pixel ZA and the second phase difference pixel ZB). Thephase difference AF can be appropriately performed even during imagingof the motion picture.

Imaging and Recording of Motion Picture

As described above, at the time of imaging and recording (displaying)the motion picture in the motion picture imaging mode, the CPU 40(second image generation unit) reads the image data from the imagingelement 16 in a thinned-out manner. In FIG. 4, the rows of the imagingelement 16 indicated by the black circle marks show one example of therows read in a thinned-out manner at the time of imaging the motionpicture.

In a case where the image data is read from the imaging element 16 in athinned-out manner, the pixels in the row direction are not thinned out.However, for the phase difference pixel row, the image processing unit24 (second image generation unit) extracts only the normal pixels at thepositions (positions indicated by the black star marks in FIG. 4) atwhich the normal pixels (G pixel or B pixel) of the phase differencepixel row are arranged as illustrated in FIG. 4. For the normal pixelrow, the image processing unit 24 calculates the average of threehorizontal pixels (three pixels having the same color) and substantiallygenerates a pixel at a horizontal thinning-out rate of 1/3. For thenormal pixel row, only the normal pixels at the positions indicated bythe black star marks may be extracted (reading at a horizontalthinning-out rate of 1/3) instead of the process of calculating theaverage of three horizontal pixels.

A plurality of two-dimensionally arranged pixels constituting one frameimage of the motion picture (second image) read in a thinned-out mannerhave the arrangement of the color filters in the Bayer arrangement. Inaddition, the motion picture includes only the normal pixels, and it isnot necessary to correct the phase difference detection pixel. Thus, afalse color does not occur.

That is, while the output data of the phase difference detection pixel(the first phase difference pixel ZA and the second phase differencepixel ZB) cannot be used as the image data of the G pixel or the B pixelof the motion picture, the plurality of pixels constituting one frameimage of the motion picture originally include the normal pixels and donot include the phase difference detection pixel as described above.Thus, it is not necessary to generate the image data of the phasedifference detection pixel by the interpolation process, and acalculation load in the image processing unit 24 can be reduced.

In addition, each frame image constituting the motion picture has asmaller image size than the still picture of full pixels. Thus, in themotion picture imaging mode, the imaging element 16 can be driven in athinned-out manner. Accordingly, low power and high speed processing canbe achieved.

Furthermore, each frame image constituting the motion picture has theBayer arrangement in the same manner as the filter arrangement of theimaging element 16. Thus, the image processing unit 24 can implement theimage processing of the motion picture in common with the imageprocessing (for example, the demosaicing process) of the still picture.

The second image generated by reading in a thinned-out manner is notlimited to the motion picture and may be a still picture having a smallimage size than the still picture of full pixels.

Interpolation Processing Unit

FIG. 7 is a block diagram illustrating an embodiment of an interpolationprocessing unit in the image processing unit 24 illustrated in FIG. 3.

An interpolation processing unit 60 illustrated in FIG. 7 is a partcorrecting (interpolating) the pixel value of the phase differencedetection pixel (the first phase difference pixel ZA and the secondphase difference pixel ZB) included in the image data (RAW data) readfrom the imaging element 16 at the time of switching to the stillpicture imaging mode and imaging the still picture (first image). Theinterpolation processing unit 60 functions as a part of the first imagegeneration unit generating the first image.

The interpolation processing unit 60 includes a gain interpolation unit62, an average value interpolation unit 64, a signal gradientcalculation unit 66, a pixel value addition unit 68, and a final pixelvalue decision unit 69.

Approximately half of the intensity of light incident on the surroundingnormal pixels is incident on the phase difference detection pixel (thefirst phase difference pixel ZA and the second phase difference pixelZB). Thus, sensitivity is decreased below that of the normal pixel, andthe phase difference detection pixel cannot be used as the normal pixel.

The gain interpolation unit 62 performs interpolation in which a signallevel is adjusted to that of the normal pixel by multiplying the pixelvalue of the phase difference detection pixel by predetermined gaininterpolation information in order to supplement the decrease insensitivity of the phase difference detection pixel.

The interpolation processing unit 60 (or the image processing unit 24)includes a gain interpolation information obtaining unit that obtainsthe gain interpolation information set for the pixel position of thein-focus pixel in the RAW data in a case where a correction target ofthe first phase difference pixel ZA or the second phase difference pixelZB is set as the in-focus pixel.

The gain interpolation information obtaining unit may calculate the gaininterpolation information corresponding to the pixel position of thein-focus pixel based on the RAW data or may obtain the gaininterpolation information from a storage unit storing the gaininterpolation information for each pixel position of the in-focus pixel.The gain interpolation information can be calculated as the ratio of thepixel value of the in-focus pixel in the RAW data to the average pixelvalue of the normal pixels having the same color and surrounding thein-focus pixel.

The average value interpolation unit 64 is a part generating the pixelvalue at the pixel position of the in-focus pixel using the pixel valuesof the surrounding pixels of the pixel position of the in-focus pixel.The average value interpolation unit 64 is provided with informationindicating a signal gradient direction calculated by the signal gradientcalculation unit 66 and the addition pixel value of the addition pixeladded by the pixel value addition unit 68.

The signal gradient calculation unit 66 calculates the signal gradientdirection in which the signal gradients of the surrounding pixels of thepixel position of the in-focus pixel are minimized in the case ofgenerating the pixel value at the pixel position of the in-focus pixel.

FIG. 8 is a diagram for describing the average value interpolation forthe phase difference detection pixel in the imaging element of the firstembodiment and illustrates the case of calculating the signal gradientdirection based on the surrounding pixels of the in-focus pixelindicated by a black circle in FIG. 8 and interpolating the pixel valueat the pixel position of the in-focus pixel.

As illustrated in FIG. 8, in the case of calculating the signal gradientdirection based on the surrounding pixels of the in-focus pixel (secondphase difference pixel ZB) indicated by the black circle, the signalgradient calculation unit 66 obtains the pixel values of the G pixelssurrounding the in-focus pixel, calculates the signal gradient of thehorizontal direction from, for example, the pixel interval between two Gpixels closest to each other in the horizontal direction (firstdirection) and the difference between the pixel values of two G pixels,and in the same manner, calculates the signal gradient of the verticaldirection (second direction), the signal gradient of a third direction(inclined upper right direction) between the first direction and thesecond direction, and the signal gradient of a fourth direction(inclined lower right direction). The signal gradient calculation unit66 calculates the direction of the minimum signal gradient among thecalculated signal gradients of the four directions as the signalgradient direction.

The use of the pixel value of the G pixel in the calculation of thesignal gradient direction is because the pixel value of the G pixelcontributes most to obtaining the brightness signal (Y) among the pixelvalues of the R pixel, the G pixel, and the B pixel. The signal gradientdirection calculated in the above manner corresponds to a directionhaving the highest correlation with the brightness among the fourdirections.

The in-focus pixel illustrated in FIG. 8 corresponds to the position ofthe B pixel. Thus, the signal gradient direction may be calculated usingthe pixel values of the B pixels surrounding the in-focus pixel, or thesignal gradient direction may be calculated using the RGB pixelssurrounding the in-focus pixel.

In the case of interpolating the pixel value at the pixel position ofthe in-focus pixel, the average value interpolation unit 64 detects aplurality of pixels that are present in the signal gradient directioncalculated by the signal gradient calculation unit 66 with the pixelposition of the in-focus pixel as a reference and have the same color asthe color at the pixel position of the in-focus pixel, and generates thepixel value at the pixel position of the in-focus pixel by interpolatingthe pixel values of the plurality of detected pixels. In the exampleillustrated in FIG. 8, the color at the pixel position of the in-focuspixel is blue (B). Thus, a plurality of B pixels present in the signalgradient direction are detected, and the pixel value at the pixelposition of the in-focus pixel is generated by interpolating the pixelvalues of the plurality of detected B pixels.

In addition, as illustrated in FIG. 8, only the normal pixel rows arearranged in upper two rows and lower two rows adjacent to the phasedifference pixel row. Thus, by using the normal pixels (B pixels) in thephase difference pixel row and the upper two rows and the lower two rows(range A of total five rows) adjacent to the phase difference pixel row,the average value interpolation unit 64 can generate the pixel value atthe pixel position of the in-focus pixel using the B pixels in anydirection (signal gradient direction) of the horizontal direction, thevertical direction, the inclined upper right direction, and the inclinedlower right direction.

In addition, by using the normal pixels in the range A of five rows, theaverage value interpolation unit 64 can interpolate the pixel value atthe pixel position of the in-focus pixel using the B pixels in adirection midway between the horizontal direction and the inclined upperright direction and a direction midway between the horizontal directionand the inclined lower right direction.

Furthermore, in a case where the range of normal pixels used in theinterpolation is increased to include the phase difference pixel row andupper four rows and lower four rows (range B of total nine rows)adjacent to the phase difference pixel row, the average valueinterpolation unit 64 can increase the number of directions in which thepixel value at the pixel position of the in-focus pixel can beinterpolated. That is, in a case where the range of normal pixels usedin the interpolation is increased to the range B of nine rows, theaverage value interpolation unit 64 can interpolate the pixel value atthe pixel position of the in-focus pixel using the B pixels in anydirection (signal gradient direction) of eight directions including fourdirections of the horizontal direction, the vertical direction, theinclined upper right direction, and the inclined lower right direction,and four directions midway among the four directions.

In this case, the signal gradient calculation unit 66 needs to calculateone direction of the eight directions as the signal gradient direction.

FIG. 9 is another diagram for describing the average value interpolationfor the phase difference detection pixel in the imaging element of thefirst embodiment and illustrates the case of calculating the signalgradient direction based on the surrounding pixels of the in-focus pixelindicated by a black circle in FIG. 9 and interpolating the pixel valueat the pixel position of the in-focus pixel.

As illustrated in FIG. 9, the in-focus pixel indicated by the blackcircle is the first phase difference pixel ZA, and the position of thein-focus pixel corresponds to the position of the G pixel. Accordingly,it is preferable that the signal gradient calculation unit 66 calculatesthe signal gradient direction by obtaining the pixel values of the Gpixels surrounding the in-focus pixel.

In addition, in the same manner as the method of interpolating the pixelvalue at the pixel position of the in-focus pixel (second phasedifference pixel) illustrated in FIG. 8, the average value interpolationunit 64 detects a plurality of pixels that are present in the signalgradient direction with the pixel position of the in-focus pixel as areference and have the same color as the color at the pixel position ofthe in-focus pixel based on the signal gradient direction calculated bythe signal gradient calculation unit 66 for the surrounding pixels ofthe in-focus pixel, and generates the pixel value at the pixel positionof the in-focus pixel by interpolating the pixel values of the pluralityof detected pixels.

In the example illustrated in FIG. 9, the color at the pixel position ofthe in-focus pixel is green (G). Thus, the average value interpolationunit 64 detects a plurality of G pixels present in the signal gradientdirection and generates the pixel value at the pixel position of thein-focus pixel by interpolating the pixel values of the plurality ofdetected G pixels. In addition, by using the normal pixels (G pixels) inthe range A of five rows including the phase difference pixel row, theaverage value interpolation unit 64 can generate (interpolate) the pixelvalue at the pixel position of the in-focus pixel using the G pixels inany direction of the horizontal direction, the vertical direction, theinclined upper right direction, and the inclined lower right direction.Furthermore, in a case where the range of normal pixels used in theinterpolation is increased to the range B of nine rows including thephase difference pixel row, the average value interpolation unit 64, byusing the normal pixels (G pixels) in the range B of nine rows, caninterpolate the pixel value at the pixel position of the in-focus pixelusing the G pixels in any direction of eight directions including fourdirections of the horizontal direction, the vertical direction, theinclined upper right direction, and the inclined lower right direction,and four directions midway among the four directions.

Next, a case where the pair of the first phase difference pixel ZA andthe second phase difference pixel ZB is used in the average valueinterpolation will be described.

In FIG. 7, the pixel value addition unit 68 obtains the addition pixelvalue of the addition pixel by adding the pixel values of the pair ofthe first phase difference pixel ZA and the second phase differencepixel ZB.

As described using FIG. 5, in a case where the pixel values of the pairof the first phase difference pixel ZA and the second phase differencepixel ZB are added, the added pixel value (addition pixel value) isequal to the pixel value of the normal pixel (G pixel) at the same pixelposition. In addition, the added pixel (addition pixel) can be regardedas being midway between the pair of the first phase difference pixel ZAand the second phase difference pixel ZB.

The addition pixel value of the addition pixel is a value close to thepixel value of the G pixel at the same pixel position and does not matchthe pixel value of the G pixel in a strict sense. Therefore, it ispreferable to adjust the level of the addition pixel value bymultiplying the addition pixel value by the level adjusting coefficientof the addition pixel that is preset or calculated by analyzing theimage data.

In a case where the color at the pixel position of the in-focus pixel isgreen (G), the average value interpolation unit 64 interpolates thepixel value of the in-focus pixel using the addition pixel value of theaddition pixel added by the pixel value addition unit as the pixel valueof one of the surrounding pixels of the in-focus pixel.

In a case where the normal pixels surrounding the in-focus pixel aresaturated, or, for example, in the case of an image in which thesurrounding area of the in-focus pixel is even, it is preferable not touse the addition pixel value in the average value interpolation.

In addition, in the case of the image in which the surrounding area ofthe in-focus pixel is even, the signal gradient calculation unit 66 candetermine that the signal gradient direction is not present. In a casewhere the signal gradient direction is not calculated by the signalgradient calculation unit 66, the average value interpolation unit 64can set the pixel value at the pixel position of the in-focus pixel tobe the pixel value of the pixel closest to the pixel position of thein-focus pixel and having the same color, or the average value of pixelvalues of a plurality of pixels having the same color in theinterpolation range with the pixel position of the in-focus pixel as areference.

The final pixel value decision unit 69 decides the final pixel value atthe pixel position of the in-focus pixel by selecting any one of pixelvalues including the pixel value interpolated by the gain interpolationunit 62 and the pixel value interpolated by the average valueinterpolation unit 64 or generating a pixel value obtained by weightedaddition of two pixel values. For example, in a case where the imagesurrounding the in-focus pixel is even, the pixel value obtained by theaverage value interpolation is preferred. In a case where the spatialfrequency of the image surrounding the in-focus pixel is high, the pixelvalue obtained by the gain interpolation is preferred. In addition, in aregion out of focus, the pixel value obtained by the average valueinterpolation is preferred.

As described above, the interpolation processing unit 60 corrects(interpolates) the pixel value of the phase difference detection pixelincluded in the RAW data read from the imaging element 16 at the time ofimaging the still picture. Accordingly, the RAW data of the stillpicture in which the pixel value at the pixel position of the phasedifference detection pixel is corrected is generated.

Second Embodiment of Imaging Element

FIG. 10 is a diagram illustrating a second embodiment of color filterarrangement and arrangement of the phase difference detection pixels inthe imaging element 16.

As illustrated in FIG. 10, the first periodic color arrangement of the Rfilter, the G filter, and the B filter arranged in each pixel of theimaging element 16 has a basic arrangement pattern P of 4×4 pixelsillustrated by a bold box. The basic arrangement pattern P is adjacentlyarranged in the horizontal direction and the vertical direction.

The basic arrangement pattern P is arranged such that 2×2 pixelsincluding the G filters and 2×2 pixels including the R filters areadjacent to each other in the horizontal direction, and 2×2 pixelsincluding the B filters and 2×2 pixels including the G filters areadjacent to each other in the horizontal direction.

The phase difference pixel row including the first phase differencepixel ZA and the second phase difference pixel ZB is periodicallyarranged in the row direction (horizontal direction) with three pixelsincluding the pair of the first phase difference pixel ZA and the secondphase difference pixel ZB and one normal pixel (in the present example,the G pixel or the B pixel) as one cycle. The phase difference pixel rowis disposed in a row (in the present example, a row in which the Gpixels and the B pixels are arranged) at an interval of a plurality ofrows.

In the imaging element 16 having the above configuration, as illustratedin FIG. 10, in a case where only the normal pixels extending in thesecond direction (column direction) from positions (positions indicatedby black star marks in FIG. 10) at which the normal pixels (G pixel or Bpixel) of the phase difference pixel row are arranged are extracted,color arrangement of the color filter of each pixel of the plurality ofextracted normal pixels is the same as the original color arrangementhaving the basic arrangement pattern P.

In addition, in the motion picture imaging mode, low power and highspeed processing can be achieved by driving the imaging element 16 in athinned-out manner. In a case where the image signal is read in athinned-out manner from the imaging element 16 at a certain row interval(rows indicated by black circle marks in FIG. 10) as illustrated in FIG.10, and only the normal pixels extending in the column direction fromthe positions at which the normal pixels of the phase difference pixelrow are arranged are extracted in an image corresponding to the imagesignal read in a thinned-out manner, the color arrangement of the colorfilter of each pixel of the plurality of extracted normal pixels is thesame as the original color arrangement having the basic arrangementpattern P.

FIG. 11 and FIG. 12 are diagrams for describing the average valueinterpolation for the phase difference detection pixel in the imagingelement of the second embodiment and illustrate the case of calculatingthe signal gradient direction based on the surrounding pixels of thein-focus pixel indicated by a black circle in FIG. 11 and FIG. 12 andinterpolating the pixel value at the pixel position of the in-focuspixel.

As illustrated in FIG. 11 and FIG. 12, by not arranging the phasedifference detection pixel in three rows above and four rows below thephase difference pixel row, the normal pixels in a range C of eight rowsincluding the phase difference pixel row can be used in the case ofinterpolating the pixel value of the phase difference detection pixelincluded in the RAW data of the still picture obtained at the time ofthe still picture imaging mode.

That is, the signal gradient calculation unit 66 illustrated in FIG. 7calculates the signal gradient direction (any direction of fourdirections including the horizontal direction, the vertical direction,the inclined upper right direction, and the inclined lower rightdirection) based on the surrounding pixels of the in-focus pixelindicated by the black circle. The average value interpolation unit 64generates (interpolates) the pixel value at the pixel position of thein-focus pixel by interpolating the pixel values of the normal pixels(normal pixels in the range C of eight rows) having the same color inthe signal gradient direction.

In the case of the color filter arrangement of the present example, thenumber of normal pixels having the same color and adjacent to thein-focus pixel is three in the case of FIG. 11 and two in the case ofFIG. 12. Thus, in a case where a normal pixel having the same color andadjacent in the calculated signal gradient direction is present, it ispreferable to use the pixel value of the adjacent normal pixel as thepixel value of the in-focus pixel.

The aspect of the imaging apparatus to which the present invention canbe applied is not limited to the imaging apparatus 10 illustrated inFIG. 1 and is exemplified by, for example, a mobile phone having acamera function, a smartphone, personal digital assistants (PDA), and aportable game console. Hereinafter, one example of the smartphone towhich the present invention can be applied will be described.

Configuration of Smartphone

FIG. 13 is a diagram illustrating the exterior of the smartphone as oneembodiment of the imaging apparatus.

A smartphone 100 illustrated in FIG. 13 includes a casing 102 having aflat plate shape. A display and input unit 120 in which a display panel121 as a display unit and an operation panel 122 as an input unit areformed as a single unit is disposed on one surface of the casing 102. Inaddition, the casing 102 comprises a speaker 131, a microphone 132, anoperation unit 140, and a camera unit 141 (imaging unit). Theconfiguration of the casing 102 is not for limitation purposes. Forexample, a configuration in which the display unit and the input unitare independently disposed can be employed, or a configuration having afolded structure or a sliding mechanism can be employed.

FIG. 14 is a block diagram illustrating an internal configuration of thesmartphone 100 illustrated in FIG. 13. As illustrated in FIG. 14, mainconstituents of the smartphone 100 comprise a wireless communicationunit 110, the display and input unit 120, a call unit 130, the operationunit 140, the camera unit 141, a storage unit 150, an externalinput-output unit 160 (output unit), a global positioning system (GPS)reception unit 170, a motion sensor unit 180, a power supply unit 190,and a main control unit 101. In addition, a main function of thesmartphone 100 comprises a wireless communication function of performingmobile wireless communication with a base station apparatus through amobile communication network.

The wireless communication unit 110 performs wireless communication withthe base station apparatus connected to the mobile communication networkin accordance with an instruction from the main control unit 101. Byusing the wireless communication, transmission and reception of variousfile data such as voice data and image data, electronic mail data, andthe like and reception of web data, streaming data, and the like areperformed.

The display and input unit 120 is a so-called touch panel comprising theoperation panel 122 arranged on the screen of the display panel 121. Thedisplay and input unit 120 visually delivers information to the user bydisplaying images (still image and motion image), text information, andthe like and detects a user operation performed on the displayedinformation under control of the main control unit 101. The operationpanel 122 is referred to as a touch panel for convenience.

The display panel 121 uses a liquid crystal display (LCD), an organicelectro-luminescence display (OELD), or the like as a display device.The operation panel 122 is a device that is disposed in a state wherethe image displayed on the display surface of the display panel 121 canbe visually recognized, and detects one or a plurality of coordinatesoperated by a finger of the user or a stylus. In a case where the deviceis operated by the finger of the user or the stylus, the operation panel122 outputs a detection signal generated by the operation to the maincontrol unit 101. Next, the main control unit 101 detects the operationposition (coordinates) on the display panel 121 based on the receiveddetection signal.

The display panel 121 and the operation panel 122 of the smartphone 100illustrated in FIG. 13 constitute the display and input unit 120 as asingle unit. The operation panel 122 is arranged to completely cover thedisplay panel 121. In the case of employing such an arrangement, theoperation panel 122 may comprise a function of detecting the useroperation even in a region outside the display panel 121. In otherwords, the operation panel 122 may comprise a detection region(hereinafter, referred to as a “display region”) for an overlapping partin overlap with the display panel 121 and a display region (hereinafter,referred to as a “non-display region”) for the other peripheral part notin overlap with the display panel 121.

The size of the display region may completely match the size of thedisplay panel 121. Both sizes do not necessarily match. In addition, theoperation panel 122 may comprise two sensitive regions including theperipheral part and the other inner part. Furthermore, the width of theperipheral part is appropriately designed depending on the size and thelike of the casing 102. Furthermore, a position detection methodemployed in the operation panel 122 is exemplified by a matrix switchmethod, a resistive film method, a surface acoustic wave method, aninfrared method, an electromagnetic induction method, an electrostaticcapacitive method, and the like. Any method may be employed.

The call unit 130 comprises the speaker 131 and the microphone 132. Thecall unit 130 converts the voice of the user input through themicrophone 132 into voice data processable in the main control unit 101and outputs the voice data to the main control unit 101, or decodes thevoice data received by the wireless communication unit 110 or theexternal input-output unit 160 and outputs the decoded voice data fromthe speaker 131. In addition, as illustrated in FIG. 13, for example,the speaker 131 and the microphone 132 can be mounted on the samesurface as the surface on which the display and input unit 120 isdisposed.

The operation unit 140 is a hardware key using a key switch or the likeand receives an instruction from the user. For example, as illustratedin FIG. 13, the operation unit 140 is a push-button type switch that ismounted on a side surface of the casing 102 of the smartphone 100. In acase where the operation unit 140 is pressed by the finger or the like,the operation unit 140 enters a switch ON state. In a case where thefinger is released, the operation unit 140 enters a switch OFF state bya restoring force of a spring or the like.

The storage unit 150 stores a control program and control data of themain control unit 101, address data in which a name, a telephone number,and the like of a communication counterpart are associated, data oftransmitted and received electronic mails, web data downloaded by webbrowsing, downloaded contents data, and the like and also temporarilystores streaming data and the like.

In addition, the storage unit 150 is composed of an internal storageunit 151 incorporated in the smartphone and an external storage unit 152including an attachable and detachable external memory slot. Each of theinternal storage unit 151 and the external storage unit 152 constitutingthe storage unit 150 is implemented using a storage medium such as amemory of a flash memory type, a hard disk type, a multimedia card microtype, or a card type, a random access memory (RAM), or a read onlymemory (ROM).

The external input-output unit 160 acts as an interface for all externalapparatuses connected to the smartphone 100 and is directly orindirectly connected to other external apparatuses by communication andthe like (for example, Universal Serial Bus (USB) and IEEE 1394) ornetworks (for example, a wireless local area network (LAN), Bluetooth(registered trademark), radio frequency identification (RFID), infrareddata association (IrDA), Ultra Wideband (UWB) (registered trademark),and ZigBee (registered trademark)).

For example, the external apparatuses connected to the smartphone 100include a wired/wireless headset, a wired/wireless external charger, awired/wireless data port, a memory card or a subscriber identity module(SIM)/user identity module (UIM) card connected through a card socket,an external audio and video apparatus connected through an audio andvideo input/output (I/O), an external audio and video apparatusconnected in a wired/wireless manner, a smartphone, a personal computer,a personal digital assistant (PDA), and an earphone. The externalinput-output unit 160 may be configured to deliver data transferred fromthe external apparatuses to each constituent inside the smartphone 100or transfer data inside the smartphone 100 to the external apparatuses.

The GPS reception unit 170 receives GPS signals transmitted from GPSsatellites ST1, ST2 to STn, executes a position measurement calculationprocess based on the plurality of received GPS signals, and obtainspositional information (GPS information) specified by the latitude, thelongitude, and the altitude of the smartphone 100 in accordance with aninstruction from the main control unit 101. In a case where thepositional information can be obtained from the wireless communicationunit 110 and/or the external input-output unit 160 (for example, awireless LAN), the GPS reception unit 170 can detect the position usingthe positional information.

The motion sensor unit 180 comprises, for example, a 3-axis accelerationsensor and detects a physical motion of the smartphone 100 in accordancewith an instruction from the main control unit 101. By detecting thephysical motion of the smartphone 100, the movement direction and theacceleration of the smartphone 100 are detected. The result of thedetection is output to the main control unit 101.

The power supply unit 190 supplies power stored in a battery (notillustrated) to each unit of the smartphone 100 in accordance with aninstruction from the main control unit 101.

The main control unit 101 comprises a microprocessor, operates inaccordance with the control program and the control data stored in thestorage unit 150, and manages and controls each unit of the smartphone100. In addition, the main control unit 101 comprises a mobilecommunication control function of controlling each unit of acommunication system and an application processing function in order toperform voice communication and data communication through the wirelesscommunication unit 110.

The application processing function is implemented by operating the maincontrol unit 101 in accordance with application software stored in thestorage unit 150. For example, the application processing functionincludes an infrared communication function of performing datacommunication with an opposing apparatus by controlling the externalinput-output unit 160, an electronic mail function of transmitting andreceiving electronic mails, and a web browsing function of browsing webpages, and also includes an image processing function according to theembodiment of the present invention.

In addition, the main control unit 101 comprises the image processingfunction such as displaying a video on the display and input unit 120based on image data (data of a still image or a motion image) such asreception data and downloaded streaming data. In addition, the imageprocessing function includes image processing performed by the imageprocessing unit 24 illustrated in FIG. 3.

Furthermore, the main control unit 101 executes display control of thedisplay panel 121 and operation detection control for detecting the useroperation performed through the operation unit 140 or the operationpanel 122.

By executing the display control, the main control unit 101 displays anicon for starting the application software or a software key such as ascroll bar, or displays a window for composing an electronic mail. Thescroll bar refers to a software key for receiving an instruction to movea display part of an image for a large image or the like notaccommodated in the display region of the display panel 121.

In addition, by executing the operation detection control, the maincontrol unit 101 detects the user operation performed through theoperation unit 140, receives an operation performed on the icon throughthe operation panel 122 or an input of a text string in an input fieldof the window, or receives a request for scrolling the display imagethrough the scroll bar.

Furthermore, by executing the operation detection control, the maincontrol unit 101 comprises a touch panel control function of determiningwhether the operation position on the operation panel 122 corresponds tothe overlapping part (display region) in overlap with the display panel121 or the other peripheral part (non-display region) not in overlapwith the display panel 121 and controlling the sensitive region of theoperation panel 122 and the display position of the software key.

In addition, the main control unit 101 can detect a gesture operationperformed on the operation panel 122 and execute a present functiondepending on the detected gesture operation. The gesture operation isnot a simple touch operation in the related art and means an operationof drawing a trajectory by the finger or the like, specifying aplurality of positions at the same time, or an operation of acombination thereof by drawing a trajectory from at least one of theplurality of positions.

The camera unit 141 converts the image data obtained by imaging intocompressed image data in, for example, Joint Photographic Experts Group(JPEG) and records the image data in the storage unit 150 or outputs theimage data through the external input-output unit 160 or the wirelesscommunication unit 110 under control of the main control unit 101. Asillustrated in FIG. 13, in the smartphone 100, the camera unit 141 ismounted on the same surface as the display and input unit 120. However,the mounting position of the camera unit 141 is not for limitationpurposes, and the camera unit 141 may not be mounted on the side surfaceof the casing 102 on which the display and input unit 120 is disposed.The camera unit 141 may be mounted on the rear surface of the casing102, or a plurality of camera units 141 may be mounted on the casing102. In a case where the plurality of camera units 141 are mounted,imaging may be performed by a single camera unit 141 by switching thecamera unit 141 performing the imaging, or imaging may be performedusing the plurality of camera units 141 at the same time.

In addition, the camera unit 141 can be used in various functions of thesmartphone 100. For example, the image obtained by the camera unit 141may be displayed on the display panel 121, or the image captured andobtained in the camera unit 141 may be used as one of operation inputmethods for the operation panel 122. In addition, in the detection ofthe position by the GPS reception unit 170, the position may be detectedwith reference to the image from the camera unit 141. Furthermore, withreference to the image from the camera unit 141, a determination of theoptical axis direction of the camera unit 141 of the smartphone 100 anda determination of the current usage environment can be performedwithout using the 3-axis acceleration sensor or along with the 3-axisacceleration sensor. The image from the camera unit 141 can also be usedin the application software.

Besides, data obtained by adding the positional information obtained bythe GPS reception unit 170, voice information (may be text informationobtained by performing voice-to-text conversion by the main control unitor the like) obtained by the microphone 132, attitude informationobtained by the motion sensor unit 180, and the like to the image dataof the still picture or the motion picture can be recorded in thestorage unit 150 or output through the external input-output unit 160 orthe wireless communication unit 110.

Others

In the imaging element of the present embodiment, while the G filter isarranged in the phase difference detection pixel, the phase differencedetection pixel may be configured such that light in a wider wavelengthrange than the transmission wavelength range of the G filter can beincident on the phase difference detection pixel. For example, atransparent filter can be used without disposing the G filter in thephase difference detection pixel. Accordingly, a high pixel value can beobtained from even the phase difference detection pixel having a smalleropening portion than the normal pixel (the phase difference detectionpixel can have high sensitivity).

In addition, in any of the first embodiment of the imaging elementillustrated in FIG. 4 and the second embodiment of the imaging elementillustrated in FIG. 10, the phase difference pixel row is configured byperiodically arranging three pixels including the pair of the firstphase difference pixel ZA and the second phase difference pixel ZB andone normal pixel as one cycle. However, the phase difference pixel rowis not for limitation purposes. For example, the phase difference pixelrow may be configured by periodically arranging five pixels includingthe pair of the first phase difference pixel ZA and the second phasedifference pixel ZB and three normal pixels. In this case, in a casewhere the normal pixels of the phase difference pixel row are read at ahorizontal thinning-out rate of 1/5, the same color arrangement as thenormal pixel row corresponding to the phase difference pixel row isachieved.

Furthermore, the present invention is not limited to the aboveembodiments, and various modifications can be made without departingfrom the spirit of the present invention.

EXPLANATION OF REFERENCES

-   1: strobe-   2: shutter button-   3: power supply/mode switch-   4: mode dial-   5: zoom button-   5T: tele button-   5W: wide button-   6: cross button-   7: MENU/OK button-   8: playback button-   9: BACK button-   10: imaging apparatus-   12: imaging lens-   14: stop-   15: mechanical shutter-   16: imaging element-   16A, 16B: light shielding member-   22: image input controller-   24: image processing unit-   26: compression/expansion processing unit-   28: video encoder-   30: liquid crystal monitor-   32: sensor drive unit-   33: shutter drive unit-   34: stop drive unit-   36: lens drive unit-   38: operation unit-   40: CPU-   42: AF processing unit-   44: AE detection unit-   47: ROM-   48: memory-   50: VRAM-   52: media controller-   54: memory card-   60: interpolation processing unit-   62: gain interpolation unit-   64: average value interpolation unit-   66: signal gradient calculation unit-   68: pixel value addition unit-   69: final pixel value decision unit-   100: smartphone-   101: main control unit-   102: casing-   110: wireless communication unit-   120: display and input unit-   121: display panel-   122: operation panel-   130: call unit-   131: speaker-   132: microphone-   140: operation unit-   141: camera unit-   150: storage unit-   151: internal storage unit-   152: external storage unit-   160: external input-output unit-   170: GPS reception unit-   180: motion sensor unit-   190: power supply unit-   CF: color filter-   L: microlens-   P: basic arrangement pattern-   PD: photodiode-   ST1: GPS satellite-   ST2: GPS satellite-   ZA: first phase difference pixel-   ZB: second phase difference pixel

What is claimed is:
 1. An imaging apparatus comprising: an imagingelement in which a plurality of phase difference detection pixels and aplurality of normal pixels for imaging are two-dimensionally arranged ina first direction and a second direction orthogonal to the firstdirection, wherein the phase difference detection pixels include a firstphase difference pixel and a second phase difference pixel which haveopening portions for pupil separation at different positions in thefirst direction, in the plurality of normal pixels, a first filtercorresponding to a first color most contributing to obtaining abrightness signal and a plurality of second filters respectivelycorresponding to two or more colors other than the first color arearranged in a first periodic color arrangement, the imaging elementincludes normal pixel rows in which only the normal pixels are arrangedin the first direction and a phase difference pixel row in which thephase difference detection pixels and the normal pixels are periodicallyarranged in the first direction, and only the normal pixel rows arearranged in at least two rows adjacent to the phase difference pixelrow; an imaging optical system forming a subject image on alight-receiving surface of the imaging element; and at least oneprocessor configured to: assuming that the first phase difference pixeland the second phase difference pixel are set as in-focus pixels,generate pixel values at pixel positions of the in-focus pixels usingpixel values of pixels around the pixel positions of the in-focuspixels, and generate a first image including the pixel values at thepixel positions of the first phase difference pixel and the second phasedifference pixel; and extract only normal pixels in the imaging elementand generate a second image comprising the extracted normal pixels,wherein the at least one processor extracts normal pixels from the phasedifference row and extracts normal pixels extending in the seconddirection from positions where the normal pixels are arranged in thephase difference pixel row, from the normal pixel rows, and generatesthe second image including the extracted normal pixels.
 2. The imagingapparatus according to claim 1, wherein the at least one processorinterpolates pixel values of normal pixels having a same color with acolor at the pixel positions of the in-focus pixels, in a case where theat least one processor generates the pixel values at the pixel positionsin the in-focus pixels in the first image.
 3. The imaging apparatusaccording to claim 1, wherein the at least one processor calculates aweighted average of pixel values of normal pixels having a same colorwith a color at the pixel positions of the in-focus pixels, in a casewhere the at least one processor generates the pixel values at the pixelpositions in the in-focus pixels in the first image.
 4. The imagingapparatus according to claim 1, wherein, in a case where the at leastone processor generates the pixel values at the pixel positions in thein-focus pixels in the first image, the at least one processorcalculates a signal gradient direction of pixels around the pixelpositions of in-focus pixels, and interpolates pixel values of pixelspresent in the signal gradient direction with respect to the pixelpositions of the in-focus pixels.
 5. The imaging apparatus according toclaim 4, wherein the at least one processor calculates as the signalgradient direction, a direction in which a signal gradient of the pixelsaround the pixel positions of the in-focus pixels is the minimum.
 6. Theimaging apparatus according to claim 4, wherein the at least oneprocessor uses normal pixels in five rows comprising the phasedifference pixel row including the in-focus pixels and two sets of twonormal pixel rows respectively adjacent to the phase difference pixelrow with the phase difference pixel row arranged between the two sets oftwo normal pixel rows, and selects as the signal gradient direction, onedirection in which the signal gradient is the minimum from among fourdirections including: the first direction; the second direction; and athird direction and a fourth direction which are between the firstdirection and the second direction, with respect to the in-focus pixels.7. The imaging apparatus according to claim 4, wherein the at least oneprocessor is further uses normal pixels in nine rows comprising thephase difference pixel row including the in-focus pixels and two sets offour normal pixel rows respectively adjacent to the phase differencepixel row with the phase difference pixel row arranged between the twosets of four normal pixel rows, and selects as the signal gradientdirection, one direction in which the signal gradient is the minimumfrom among eight directions including: the first direction; the seconddirection; a third direction and a fourth direction which are betweenthe first direction and the second direction; and four directions whichare intermediate directions of the first direction, the seconddirection, the third direction and the fourth direction.
 8. The imagingapparatus according to claim 1, further comprising a mode switchconfigured to switch a mode between a first mode for generating thefirst image and a second mode for generating the second image, whereinthe at least one processor is further configured to: generate the firstimage in a case where the mode is switched to the first mode by the modeswitch; and generate the second image in a case where the mode isswitched to the second mode by the mode switch.
 9. The imaging apparatusaccording to claim 1, wherein the first image is a still image and thesecond image is each frame image constituting a motion image.
 10. Theimaging apparatus according to claim 1, wherein a color arrangement ofthe normal pixels in the second image corresponds to a color filterhaving a periodic color arrangement identical to the first periodiccolor arrangement.
 11. The imaging apparatus according to claim 1,wherein the first periodic color arrangement is a Bayer arrangement, andin the phase difference pixel row, three pixels including a pair of thefirst phase difference pixel and the second phase difference pixel, andone normal pixel, are periodically arranged as one cycle.
 12. Theimaging apparatus according to claim 1, wherein the first filter is a Gfilter transmitting light in a wavelength range of green, and theplurality of second filters include an R filter transmitting light in awavelength range of red and a B filter transmitting light in awavelength range of blue, the first periodic color arrangement has abasic arrangement pattern of 4-by-4 pixels adjacently arranged in thefirst direction and the second direction, in the basic arrangementpattern, 2-by-2 pixels including the G filter and 2-by-2 pixelsincluding the R filter are adjacently arranged in the first direction,and 2-by-2 pixels including the B filter and 2-by-2 pixels including theG filter are adjacently arranged in the first direction, and in thephase difference pixel row, three pixels including a pair of the firstphase difference pixel and the second phase difference pixel, and onenormal pixel, are periodically arranged as one cycle.
 13. The imagingapparatus according to claim 1, further comprising: a phase differencedetector configured to detect a phase difference between a first pixelvalue obtained from the first phase difference pixel and a second pixelvalue obtained from the second phase difference pixel, in the phasedifference pixel row in the imaging element; and an autofocus controllerconfigured to control the imaging optical system based on the phasedifference detected by the phase difference detector.